What's in these flash-sorted files?

The flash sorting process produces binary files. The purpose of this notebook is to demonstrate how to read and display data from these files. In practice, these files are no more difficult to work with than columns of ASCII data, because modern access libraries automate the annoying and tedious parts of traversing the file's internal hierarchy.

The flash sorting analysis produces three kinds of files.

  • HDF5 files with two data tables
    • the VHF sources themselves
      • replicates ASCII data
      • retains single-point flashes
    • information about each flash
  • Gridded NetCDF files
    • Counts up things that are in the HDF5 files
      • source, flash, flash initiation density
      • average flash area (area-weighted flash density)
    • The grids are CF-compliant
  • Plots (PDF) of NetCDF grids
    • multi-panel per page, one panel for each time interval
    • very simple, with no maps, etc.
    • useful for a quick reference

Additional data tables and gridded fields can be added to accomodate, for instance, VLF/LF stroke detection data, and an associated LMA flash_id.


In [1]:
%%bash

RESULTS="/data/GLM-wkshp/flashsort/results/"
ls $RESULTS
echo "-----"
ls  $RESULTS/h5_files/2009/Apr/10/
echo "-----"
ls  $RESULTS/grid_files/2009/Apr/10/
echo "-----"
cat $RESULTS/h5_files/2009/Apr/10/input_params.py


2009-04-10T18:00:00_flash_extent.png
2009-04-10T18:01:00_flash_extent.png
2009-04-10T18:02:00_flash_extent.png
2009-04-10T18:03:00_flash_extent.png
2009-04-10T18:04:00_flash_extent.png
2009-04-10T18:05:00_flash_extent.png
2009-04-10T18:06:00_flash_extent.png
2009-04-10T18:07:00_flash_extent.png
2009-04-10T18:08:00_flash_extent.png
2009-04-10T18:09:00_flash_extent.png
2009-04-10T18:10:00_flash_extent.png
2009-04-10T18:11:00_flash_extent.png
2009-04-10T18:12:00_flash_extent.png
2009-04-10T18:13:00_flash_extent.png
2009-04-10T18:14:00_flash_extent.png
2009-04-10T18:15:00_flash_extent.png
2009-04-10T18:16:00_flash_extent.png
2009-04-10T18:17:00_flash_extent.png
2009-04-10T18:18:00_flash_extent.png
2009-04-10T18:19:00_flash_extent.png
2009-04-10T18:20:00_flash_extent.png
2009-04-10T18:21:00_flash_extent.png
2009-04-10T18:22:00_flash_extent.png
2009-04-10T18:23:00_flash_extent.png
2009-04-10T18:24:00_flash_extent.png
2009-04-10T18:25:00_flash_extent.png
2009-04-10T18:26:00_flash_extent.png
2009-04-10T18:27:00_flash_extent.png
2009-04-10T18:28:00_flash_extent.png
2009-04-10T18:29:00_flash_extent.png
2009-04-10T18:30:00_flash_extent.png
2009-04-10T18:31:00_flash_extent.png
2009-04-10T18:32:00_flash_extent.png
2009-04-10T18:33:00_flash_extent.png
2009-04-10T18:34:00_flash_extent.png
2009-04-10T18:35:00_flash_extent.png
2009-04-10T18:36:00_flash_extent.png
2009-04-10T18:37:00_flash_extent.png
2009-04-10T18:38:00_flash_extent.png
2009-04-10T18:39:00_flash_extent.png
2009-04-10T18:40:00_flash_extent.png
2009-04-10T18:41:00_flash_extent.png
2009-04-10T18:42:00_flash_extent.png
2009-04-10T18:43:00_flash_extent.png
2009-04-10T18:44:00_flash_extent.png
2009-04-10T18:45:00_flash_extent.png
2009-04-10T18:46:00_flash_extent.png
2009-04-10T18:47:00_flash_extent.png
2009-04-10T18:48:00_flash_extent.png
2009-04-10T18:49:00_flash_extent.png
2009-04-10T18:50:00_flash_extent.png
2009-04-10T18:51:00_flash_extent.png
2009-04-10T18:52:00_flash_extent.png
2009-04-10T18:53:00_flash_extent.png
2009-04-10T18:54:00_flash_extent.png
2009-04-10T18:55:00_flash_extent.png
2009-04-10T18:56:00_flash_extent.png
2009-04-10T18:57:00_flash_extent.png
2009-04-10T18:58:00_flash_extent.png
2009-04-10T18:59:00_flash_extent.png
Flash-autosort.log
grid_files
h5_files
plots
-----
LYLOUT_090410_160000_3600.dat.flash.h5
LYLOUT_090410_170000_3600.dat.flash.h5
LYLOUT_090410_180000_3600.dat.flash.h5
LYLOUT_090410_190000_3600.dat.flash.h5
input_params.py
-----
NALMA_20090410_160000_3600_10src_0.0109deg-dx_flash_extent.nc
NALMA_20090410_160000_3600_10src_0.0109deg-dx_flash_init.nc
NALMA_20090410_160000_3600_10src_0.0109deg-dx_footprint.nc
NALMA_20090410_160000_3600_10src_0.0109deg-dx_source.nc
NALMA_20090410_170000_3600_10src_0.0109deg-dx_flash_extent.nc
NALMA_20090410_170000_3600_10src_0.0109deg-dx_flash_init.nc
NALMA_20090410_170000_3600_10src_0.0109deg-dx_footprint.nc
NALMA_20090410_170000_3600_10src_0.0109deg-dx_source.nc
NALMA_20090410_180000_3600_10src_0.0109deg-dx_flash_extent.nc
NALMA_20090410_180000_3600_10src_0.0109deg-dx_flash_init.nc
NALMA_20090410_180000_3600_10src_0.0109deg-dx_footprint.nc
NALMA_20090410_180000_3600_10src_0.0109deg-dx_source.nc
NALMA_20090410_190000_3600_10src_0.0109deg-dx_flash_extent.nc
NALMA_20090410_190000_3600_10src_0.0109deg-dx_flash_init.nc
NALMA_20090410_190000_3600_10src_0.0109deg-dx_footprint.nc
NALMA_20090410_190000_3600_10src_0.0109deg-dx_source.nc
-----
{'merge_critical_time': 0.5, 'chi2': (0, 1.0), 'thresh_duration': 3.0, 'ctr_lat': 34.72461, 'lat': (-90.0, 90.0), 'alt': (0.0, 20000.0), 'mask_length': 4, 'ascii_flashes_out': 'flashes_out.dat', 'distance': 3000.0, 'stations': (6, 13), 'lon': (-180.0, 180.0), 'thresh_critical_time': 0.15, 'ctr_lon': -86.64533}

Inside the HDF5 files

To look inside the HDF5 file, we'll use the Python library known as PyTables.

The file object itself can be printed to describe the file's contents.


In [2]:
import os
import tables, pandas
import numpy as np

results_dir = "/data/GLM-wkshp/flashsort/results/"
h5name = os.path.join(results_dir, "h5_files/2009/Apr/10/LYLOUT_090410_180000_3600.dat.flash.h5")
h5 = tables.open_file(h5name)
print h5


/data/GLM-wkshp/flashsort/results/h5_files/2009/Apr/10/LYLOUT_090410_180000_3600.dat.flash.h5 (File) 'Flash-sorted New Mexico Tech LMA Data'
Last modif.: 'Wed Sep  2 19:59:45 2015'
Object Tree: 
/ (RootGroup) 'Flash-sorted New Mexico Tech LMA Data'
/NLDN (Group) 'NLDN strokes'
/NLDN/NLDN_090410_180000_600 (Table(0,)) 'NLDN_090410_180000_600'
/events (Group) 'Analyzed detected events'
/events/LMA_090410_180000_3600 (Table(2297208,)) 'LMA_090410_180000_3600'
/flashes (Group) 'Sorted LMA flash data'
/flashes/LMA_090410_180000_3600 (Table(139582,)) 'LMA_090410_180000_3600_cg'

The file's internal hierarchy of groups and data tables is easily navigated by tab-completion. In the next cell, type

h5.root.

and then tab, and you'll get a list of group and table names.

Let's look at events first. tab-complete to that group and then tab-complete again to get a reference to the data table. Note that a flash_id column has been added to the original source data.


In [3]:
h5.root.


  File "<ipython-input-3-d6a4c72d774a>", line 1
    h5.root.
            ^
SyntaxError: invalid syntax

In [4]:
event_table = h5.root.events.LMA_090410_180000_3600
event_table


Out[4]:
/events/LMA_090410_180000_3600 (Table(2297208,)) 'LMA_090410_180000_3600'
  description := {
  "alt": Float32Col(shape=(), dflt=0.0, pos=0),
  "charge": Int8Col(shape=(), dflt=0, pos=1),
  "chi2": Float32Col(shape=(), dflt=0.0, pos=2),
  "flash_id": Int32Col(shape=(), dflt=0, pos=3),
  "lat": Float32Col(shape=(), dflt=0.0, pos=4),
  "lon": Float32Col(shape=(), dflt=0.0, pos=5),
  "mask": StringCol(itemsize=4, shape=(), dflt='', pos=6),
  "power": Float32Col(shape=(), dflt=0.0, pos=7),
  "stations": UInt8Col(shape=(), dflt=0, pos=8),
  "time": Float64Col(shape=(), dflt=0.0, pos=9)}
  byteorder := 'little'
  chunkshape := (1724,)

We can get all the altitudes ...


In [5]:
event_table.cols.alt[:]


Out[5]:
array([ 4374.60009766,  5411.29980469,  6541.10009766, ...,  5360.39990234,
        5398.89990234,  6691.20019531], dtype=float32)

Or the first ten sources ...


In [6]:
event_table[:10]


Out[6]:
array([ (4374.60009765625, 0, 0.05999999865889549, 0, 35.799869537353516, -86.26461029052734, '015d', 14.300000190734863, 6, 64803.189990995),
       (5411.2998046875, 0, 0.07999999821186066, 0, 35.768680572509766, -86.27728271484375, '015d', 13.300000190734863, 6, 64803.036388577),
       (6541.10009765625, 0, 0.15000000596046448, 0, 35.734657287597656, -86.29420471191406, '01dd', 15.300000190734863, 7, 64803.033087856),
       (5211.89990234375, 0, 0.05000000074505806, 0, 35.77305603027344, -86.30245971679688, '015d', 15.199999809265137, 6, 64803.000147067),
       (5158.7998046875, 0, 0.10999999940395355, 0, 35.79684829711914, -86.25887298583984, '05fd', 19.100000381469727, 9, 64803.197303511),
       (3234.89990234375, 0, 0.550000011920929, 0, 35.77531051635742, -86.29548645019531, '0777', 25.600000381469727, 9, 64803.196962262),
       (4992.10009765625, 0, 0.5600000023841858, 0, 35.769142150878906, -86.29806518554688, '03dd', 15.699999809265137, 8, 64803.101830194),
       (4032.5, 0, 0.33000001311302185, 0, 35.80069351196289, -86.2587661743164, '035d', 17.600000381469727, 7, 64803.19253486),
       (4656.39990234375, 0, 0.47999998927116394, 0, 35.78547668457031, -86.29573822021484, '015d', 11.5, 6, 64802.952142755),
       (4905.0, 0, 0.12999999523162842, 0, 35.799156188964844, -86.26167297363281, '01dd', 15.699999809265137, 7, 64803.191453212)], 
      dtype=[('alt', '<f4'), ('charge', 'i1'), ('chi2', '<f4'), ('flash_id', '<i4'), ('lat', '<f4'), ('lon', '<f4'), ('mask', 'S4'), ('power', '<f4'), ('stations', 'u1'), ('time', '<f8')])

... or the flash IDs for the first hundred sources.


In [7]:
event_table[:100]['flash_id']


Out[7]:
array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
       0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
       1, 1, 1, 1, 1, 1, 1, 1], dtype=int32)

Let's look at flash #27. The indexing operation (colon) is what actually lets us access the data, instead of just the description of the data column.


In [35]:
print event_table.cols.flash_id
query = (event_table.cols.flash_id[:] == 27)
fl_27_events = event_table[query][:]
lats, lons, alts, times = fl_27_events['lat'], fl_27_events['lon'], fl_27_events['alt'], fl_27_events['time']
for lat, lon in zip(lats, lons):
    print lat, lon


/events/LMA_090410_180000_3600.cols.flash_id (Column(2297208,), int32, idx=None)
35.1792 -87.2165
35.1794 -87.2334
35.1744 -87.2277
35.1816 -87.2336
35.1763 -87.2321
35.1795 -87.2339
35.1774 -87.2355
35.1803 -87.2326
35.1773 -87.2347
35.178 -87.2365
35.1831 -87.2346
35.1771 -87.2301
35.1834 -87.2371
35.1839 -87.2334
35.1777 -87.2299
35.1843 -87.2323
35.1703 -87.2322
35.1757 -87.2392
35.1801 -87.2437
35.1867 -87.231
35.1752 -87.2347
35.1747 -87.2325
35.1887 -87.2515
35.1809 -87.228
35.1843 -87.2236
35.1673 -87.2166
35.166 -87.2137
35.1865 -87.2299
35.1819 -87.2218
35.1841 -87.2303
35.1851 -87.2341
35.1802 -87.228
35.1812 -87.2285
35.1468 -87.2097
35.1831 -87.2244
35.1808 -87.2223
35.1801 -87.2205
35.2099 -87.1962
35.1552 -87.2193
35.2192 -87.2035
35.181 -87.2211
35.2189 -87.2046
35.2126 -87.198
35.1838 -87.2267
35.2043 -87.1903
35.2201 -87.2055
35.2146 -87.2043
35.185 -87.2235
35.195 -87.1844
35.214 -87.2072
35.186 -87.2263
35.1281 -87.2041
35.2164 -87.2109
35.2124 -87.2098
35.1945 -87.1888
35.1927 -87.188
35.2156 -87.2167
35.1495 -87.2143
35.1879 -87.2266
35.1758 -87.2106
35.1895 -87.2295
35.1823 -87.2299
35.1745 -87.2415
35.2175 -87.1966
35.1889 -87.2317
35.1879 -87.2314
35.1929 -87.1874
35.1966 -87.195
35.1824 -87.1614
35.1913 -87.1762
35.2006 -87.1863
35.1955 -87.182
35.1987 -87.1976
35.1896 -87.1775
35.1951 -87.1922
35.1951 -87.23
35.2023 -87.1856
35.2024 -87.1971
35.1994 -87.1944
35.2051 -87.1903
35.1996 -87.1915
35.2035 -87.1999
35.2079 -87.2121
35.208 -87.2121
35.2025 -87.2062
35.1877 -87.1792
35.2047 -87.2068
35.2009 -87.2029
35.2111 -87.2125
35.2046 -87.2061
35.2037 -87.2048
35.2064 -87.203
35.2031 -87.1989
35.2046 -87.1997
35.199 -87.1922
35.2029 -87.1801
35.1942 -87.1887
35.2047 -87.1812
35.1872 -87.2091
35.1948 -87.2392
35.186 -87.214
35.1823 -87.21
35.202 -87.2292
35.1925 -87.2215
35.1888 -87.212
35.1942 -87.2129
35.18 -87.1908
35.1872 -87.2041
35.1869 -87.1954
35.1932 -87.1978
35.211 -87.1755
35.196 -87.1965
35.1944 -87.1868
35.1874 -87.2006
35.1869 -87.1874
35.2186 -87.1885
35.1999 -87.1994
35.1799 -87.2115
35.1933 -87.1917
35.1857 -87.1804
35.2134 -87.1856
35.2057 -87.1763
35.2078 -87.1797
35.2072 -87.1814
35.2045 -87.1767
35.1745 -87.1567
35.2015 -87.1753
35.1913 -87.236
35.2196 -87.1967
35.199 -87.1875
35.1795 -87.215
35.138 -87.2197
35.138 -87.2315
35.2042 -87.203
35.1467 -87.2274
35.1849 -87.2209
35.1836 -87.2214
35.1841 -87.2221
35.2106 -87.1832
35.1742 -87.2076
35.2088 -87.1879
35.1706 -87.2009
35.2244 -87.1996
35.1715 -87.2071
35.1496 -87.2142
35.2215 -87.205
35.1822 -87.2195

In [36]:
import matplotlib
matplotlib.rc('font', size=12)
%matplotlib inline

import matplotlib.pyplot as plt
plt.scatter(lons, lats, c=times)
plt_range =  (lons.min(),lons.max(),lats.min(),lats.max())
print plt_range
plt.axis(plt_range)


(-87.251526, -87.156723, 35.12809, 35.224358)
Out[36]:
(-87.251526, -87.156723, 35.12809, 35.224358)

Now, let's get the associated flash properties. Use pandas to make an appealing print of the the data table.


In [13]:
flash_table = h5.root.flashes.LMA_090410_180000_3600
fl_27 = flash_table[flash_table.cols.flash_id[:] == 27] # This is a numpy array with a named dtype
fl_27 = pandas.DataFrame(fl_27)

In [14]:
fl_27


Out[14]:
area ctr_alt ctr_lat ctr_lon duration flash_id init_alt init_lat init_lon init_pts n_points start CG
0 56.588226 5765.664062 35.190117 -87.209343 0.439238 27 6885.399902 35.184879 -87.220871 147 64801.154587 False

In [15]:
fl_27_events = pandas.DataFrame(fl_27_events)
fl_27_events


Out[15]:
alt charge chi2 flash_id lat lon mask power stations time
0 5842.799805 0 0.62 27 35.179199 -87.216492 019d 7.600000 6 64801.205053
1 5592.700195 0 0.26 27 35.179440 -87.233368 07df 14.000000 10 64801.229406
2 5198.399902 0 0.59 27 35.174419 -87.227669 079c 10.900000 7 64801.229690
3 5450.500000 0 0.23 27 35.181629 -87.233559 025d 9.700000 6 64801.230050
4 4926.000000 0 0.45 27 35.176300 -87.232147 07cd 11.300000 8 64801.230588
5 5968.399902 0 0.61 27 35.179451 -87.233910 05f7 14.600000 9 64801.230748
6 6123.600098 0 0.22 27 35.177376 -87.235466 03c9 11.000000 6 64801.230961
7 5068.399902 0 0.23 27 35.180313 -87.232552 07dd 12.700000 9 64801.231316
8 4943.500000 0 0.73 27 35.177345 -87.234688 07dd 16.600000 9 64801.231521
9 5369.600098 0 0.48 27 35.177967 -87.236488 079d 13.600000 8 64801.231750
10 5142.899902 0 0.35 27 35.183121 -87.234581 07dd 15.900000 9 64801.233874
11 5222.299805 0 0.81 27 35.177063 -87.230110 013d 10.700000 6 64801.234400
12 4791.700195 0 0.56 27 35.183365 -87.237122 037d 10.200000 8 64801.234625
13 5139.299805 0 0.14 27 35.183918 -87.233368 03d8 8.700000 6 64801.236230
14 5522.200195 0 0.48 27 35.177654 -87.229851 0759 14.100000 7 64801.229130
15 4831.399902 0 0.69 27 35.184330 -87.232292 015d 9.900000 6 64801.236556
16 5638.100098 0 0.65 27 35.170288 -87.232231 011f 11.000000 6 64801.434383
17 5983.399902 0 0.45 27 35.175709 -87.239151 02cf 12.700000 7 64801.237968
18 4889.399902 0 0.80 27 35.180058 -87.243690 03c7 5.400000 7 64801.239572
19 5076.799805 0 0.36 27 35.186699 -87.231010 035f 12.100000 8 64801.240193
20 5925.600098 0 0.98 27 35.175201 -87.234665 07cf 8.200000 9 64801.423108
21 5087.799805 0 0.45 27 35.174660 -87.232513 035d 4.300000 7 64801.422582
22 6887.799805 0 0.89 27 35.188744 -87.251526 021f 11.500000 6 64801.421703
23 5996.500000 0 0.09 27 35.180893 -87.228027 04d6 21.700001 6 64801.419158
24 6624.399902 0 0.46 27 35.184338 -87.223618 0693 19.700001 6 64801.419029
25 2832.300049 0 0.50 27 35.167282 -87.216599 03dd 10.900000 8 64801.417423
26 4397.700195 0 0.93 27 35.166023 -87.213692 036b 6.400000 7 64801.416971
27 4969.899902 0 0.56 27 35.186546 -87.229904 015d 9.600000 6 64801.240721
28 2780.800049 0 0.52 27 35.181892 -87.221756 016d 5.900000 6 64801.242366
29 3929.899902 0 0.16 27 35.184113 -87.230324 034d 8.600000 6 64801.237740
... ... ... ... ... ... ... ... ... ... ...
117 4065.199951 0 0.75 27 35.179886 -87.211510 039d 7.900000 7 64801.359373
118 6835.000000 0 0.87 27 35.193348 -87.191689 03cf 9.000000 8 64801.298402
119 6441.200195 0 0.49 27 35.185730 -87.180351 05d6 12.300000 7 64801.298738
120 6811.700195 0 0.56 27 35.213352 -87.185608 03cd 8.600000 7 64801.354133
121 6039.500000 0 0.83 27 35.205727 -87.176331 075f 12.200000 9 64801.353956
122 5358.899902 0 0.80 27 35.207767 -87.179703 035d 11.100000 7 64801.353567
123 6036.899902 0 0.86 27 35.207218 -87.181427 01df 10.900000 8 64801.353197
124 6538.299805 0 0.49 27 35.204475 -87.176682 07df 15.600000 10 64801.349032
125 5296.200195 0 0.70 27 35.174534 -87.156723 01ea 10.000000 6 64801.348935
126 5988.500000 0 0.92 27 35.201550 -87.175285 079f 16.799999 9 64801.348452
127 5501.399902 0 0.87 27 35.191257 -87.235992 034d 4.400000 6 64801.251554
128 5114.299805 0 0.80 27 35.219639 -87.196701 07dd 11.600000 9 64801.498893
129 6262.700195 0 0.44 27 35.199043 -87.187515 07d5 14.200000 8 64801.307827
130 7465.299805 0 0.47 27 35.179455 -87.214981 031f 10.900000 7 64801.177035
131 2267.399902 0 0.84 27 35.137993 -87.219688 030f 6.700000 6 64801.566668
132 3920.500000 0 0.89 27 35.138016 -87.231537 03db 11.400000 8 64801.593825
133 6507.500000 0 0.35 27 35.204247 -87.202957 068f 11.100000 7 64801.307996
134 2248.699951 0 0.91 27 35.146744 -87.227394 035d 10.300000 7 64801.560333
135 6885.399902 0 0.23 27 35.184879 -87.220871 015f 10.100000 7 64801.154587
136 6930.399902 0 0.15 27 35.183624 -87.221420 07df 11.600000 10 64801.157741
137 6694.100098 0 0.54 27 35.184093 -87.222099 034b 3.500000 6 64801.157905
138 5557.700195 0 0.90 27 35.210648 -87.183228 035d 9.600000 7 64801.547042
139 6651.399902 0 0.57 27 35.174229 -87.207596 01cd 11.000000 6 64801.165079
140 3989.100098 0 0.11 27 35.208755 -87.187881 01cd 9.900000 6 64801.509494
141 3160.600098 0 0.27 27 35.170624 -87.200851 026d 9.600000 6 64801.165449
142 5809.100098 0 0.02 27 35.224358 -87.199646 02ce 9.200000 6 64801.501821
143 6635.600098 0 0.16 27 35.171528 -87.207123 014f 4.900000 6 64801.169525
144 3861.199951 0 0.93 27 35.149612 -87.214172 06fb 16.600000 9 64801.508590
145 5936.799805 0 0.59 27 35.221455 -87.204956 07df 19.100000 10 64801.509117
146 6440.799805 0 0.36 27 35.182205 -87.219482 034d 6.200000 6 64801.158235

147 rows × 10 columns

More sophisticated queries with the HDF5 data (and pandas) are possible.

Inside the NetCDF files


In [16]:
import netCDF4 as ncdf

In [17]:
nc_name = os.path.join(results_dir, "grid_files/2009/Apr/10/NALMA_20090410_180000_3600_10src_0.0109deg-dx_flash_extent.nc")
nc = ncdf.Dataset(nc_name)
print nc
print nc.variables['crs']


<type 'netCDF4._netCDF4.Dataset'>
root group (NETCDF3_CLASSIC data model, file format UNDEFINED):
    dimensions(sizes): lon(400), lat(400), ntimes(60)
    variables(dimensions): int32 crs(), float32 time(ntimes), float32 longitude(lon), float32 latitude(lat), int32 flash_extent(ntimes,lon,lat)
    groups: 

<type 'netCDF4._netCDF4.Variable'>
int32 crs()
    semi_major_axis: 6.37814e+06
    longitude_of_prime_meridian: 0.0
    grid_mapping_name: latitude_longitude
    inverse_flattening: 298.257
unlimited dimensions: 
current shape = ()
filling off


In [20]:
from lmatools.multiples_nc import centers_to_edges
matplotlib.rc('xtick', labelsize=10) 
matplotlib.rc('ytick', labelsize=10) 
lon, lat, fl_dens = nc.variables['longitude'][:], nc.variables['latitude'][:], nc.variables['flash_extent'][:]
lon_edge, lat_edge = centers_to_edges(lon), centers_to_edges(lat)

In [21]:
time_index = 0
plt.pcolormesh(lon_edge, lat_edge, fl_dens[time_index,:,:], cmap='gray_r')
cbar = plt.colorbar()


Saving a custom animation

The lmatools package includes a way to access these grids as a collection. We will use the collection to make a series of frames for an animation. This example is adapted from a notebook included in lmatools.


In [27]:
import glob
from lmatools.grid_collection import LMAgridFileCollection

nc_filenames=glob.glob(os.path.join(results_dir, "grid_files/2009/Apr/10/*1[89]00*_flash_extent.nc"))
print nc_filenames
nc_field = 'flash_extent'
NCs = LMAgridFileCollection(nc_filenames, nc_field, x_name='longitude', y_name='latitude')


['/data/GLM-wkshp/flashsort/results/grid_files/2009/Apr/10/NALMA_20090410_180000_3600_10src_0.0109deg-dx_flash_extent.nc', '/data/GLM-wkshp/flashsort/results/grid_files/2009/Apr/10/NALMA_20090410_190000_3600_10src_0.0109deg-dx_flash_extent.nc']

In [28]:
from datetime import datetime
t  = datetime(2009,4,10,18,30,00)
xedge, yedge, data = NCs.data_for_time(t)
limits = xedge.min(), xedge.max(), yedge.min(), yedge.max()

In [29]:
fig = plt.figure(figsize=(12,12))
if False: # set this to True to actually write files.
    for t, lon, lat, data in NCs:
        ax = plt.subplot(111)
        mesh = ax.pcolormesh(lon, lat, np.log10(data), vmin=0, vmax=3, cmap='gray_r')
        ax.axis(limits)
        plt.colorbar(mesh)
        outfile = "{0}_{1}.png".format(t.isoformat(), nc_field)
        save_path = os.path.join(results_dir, outfile)
        print(save_path)
        fig.savefig(save_path)
        fig.clf()


<matplotlib.figure.Figure at 0x118068690>

Interactive animation

Even better, we can use the notebook's widget system to make an interactive version of the above.


In [30]:
from IPython.html.widgets import interactive
from IPython.display import display

In [32]:
from matplotlib.colors import LogNorm

def plot_for_frame(frame=0):
    # get the data
    t = NCs.times[frame]
    xedge, yedge, data = NCs.data_for_time(t)
    
    # plot a frame of data
    fig = plt.figure(figsize=(10,10))
    ax = fig.add_subplot(111)
    mesh = ax.pcolormesh(xedge, yedge, data, 
                         vmin=1, vmax=100, norm=LogNorm(), cmap='gray_r')
    ax.axis(limits)
    plt.colorbar(mesh, ax=ax)
    title = "{1} at {0}".format(t.isoformat(), nc_field)
    ax.set_title(title)
    return fig
    
N_frames = len(NCs.times)
w = interactive(plot_for_frame, frame=(0, N_frames-1))
display(w)



In [25]:


In [ ]:


In [ ]:


In [ ]:


In [ ]: